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ABSTRACT 

Program and professional development staff in adult education 
currently employ a variety of strategies to select professional development 
materials, but generally lack a systematic selection approach. Selections are 
often based on familiarity rather than quality or how well-suited the 
materials are to the learning goals of instructors and the objectives of 
programs. In order to develop a framework for programs to assess the 
viability and appropriateness of resources, and to guide the development of 
new professional development materials, selection and development criteria 
were created using a field-based research approach that included a selected 
literature review, field surveys, interviews, and reviews by experts in the 
field. Twenty criteria that fall into the following four distinct categories 
were identified: (1) appropriateness of general content; (2) appropriateness 

of design and delivery; (3) quality of research base; and (4) ease of 
adoption/adaptation. Each category has a set of criteria as well as 
indicators or descriptors that more fully explain the criteria. Adult 
education professional development staff and program administrators at the 
national, state, regional, and local levels can use the criteria to help them 
both select existing professional development materials and develop their 
own. (Appendices include selection and development criteria and a rubric 
developed by the California Adult Literacy Professional Development Project. 
Contains 29 references and a list of telephone interviews.) (MO) 
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INTRODUCTION 



A primary goal of “Building Professional Development Partnerships for Adult Educators 
(PRO-NET 2000),” an American Institutes for Research® project funded by the U.S. Department of 
Education, Division of Adult Education and Literacy, is to enhance the quality of professional 
development at the State and local level. PRO-NET 2000 provides State and local administrators and 
professional development coordinators with current, research-based resources to support the design 
and implementation of comprehensive professional development systems. This publication, 
“Evaluating Professional Development Resources: Selection and Development Criteria,” is designed 
to help the field assess the quality of resources and materials used in professional development for 
instructors and administrators and to provide guidelines for the development of quality resources. 
Such resources may include train-the-trainer modules, how-to guides, e-leaming courses, 
monographs, and research reports. Some resources such as the train-the-trainer modules and e- 
leaming courses constitute the actual professional development; others provide supporting 
information. 

Many professional development resources are currently available. Check almost any Web site 
that focuses on professional development and you will find resources on a wide range of topics. This 
raises a number of questions for professional development staff such as: Which resources shall we 
choose? Which ones meet the needs of our staff? Will they improve instructional practices? What is 
the evidence that these resources are effective in meeting program and instructional goals? Are they 
aligned with State and/or district standards? 

Program and professional development staff currently employ a variety of strategies to select 
professional development materials, but generally lack a systematic approach to selecting such 
materials. Strategies include seeking recommendations from colleagues in the field, assessing the 
cost and availability of materials, and returning to familiar publishers or authors. With limited time 
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and funds it is often difficult for a program to thoroughly evaluate a resource. As a result, selections 
are often made based on familiarity rather than quality or how well-suited the materials are to the 
learning goals of the instructors and the objectives of the program. The fact that the majority of adult 
education instructors work part-time, have a high rate of turnover, come from the K- 1 2 arena with 
little knowledge of how adults learn, and lack certification in adult education presents a challenge in 
providing professional development. This challenge heightens the necessity for a more systematic 
and uniform approach to evaluating and selecting professional development resources. The purpose 
of this document is to develop a framework for programs to assess the viability and appropriateness 
of resources, and to guide the development of new professional development materials. 

A review of literature shows that while the field of education moves toward evidence-based 
research to identify “what works,” little has been done to extend this effort to professional 
development. For example, there is a large body of literature in the K— 12 arena that draws on expert 
experiences to determine “best practices” for professional development, but relatively little 
systematic research has been conducted on the type of professional development necessary to 
improve instructional practices or learner outcomes. This holds true for adult education as well. 
Although there have been no scientifically-based studies on “what works,” several qualitative studies 
have been conducted that address the issue of teacher change after participation in professional 
development. 

The National Center for the Study of Adult Learning and Literacy (NCSALL), for example, 
studied 100 New England teachers who participated in up to 18 hours of professional development 
on the topics of learner motivation, retention, and persistence, in one of three models: multi-session 
workshops, mentor-teacher groups, and practitioner research groups. The study found that teachers 
were not impacted as much by the model of professional development in which they participated as 
by other factors. The factors that impacted teacher change included the amount of time spent on the 
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professional development, the quality of the professional development, the teacher’s personal 
motivation for learning, the program’s working conditions, and also the program’s structure. These 
factors are similar to those that have been found to impact adult learners, and therefore, the results of 
such studies can be reasonably applied to the effects of professional development on instructional 
practices and learner outcomes within adult education. 

Another research study gathered data from practitioners about how they define the impact of 
professional development. While improvement of learner outcomes is a key factor, the study found 
that impact was defined along a continuum ranging from “changes in classroom practice, to ideas and 
theories about teaching and learning that drive practice, to organizational, structural, and conceptual 
issues at the program level and in the broader field of adult education’’ (Belzer, 2003). Data collected 
also showed that different visions for professional development had different kinds of impact. 

While these studies do not provide data on the quality of resources needed to support 
effective professional development practices, the literature does identify key elements of quality 
professional development resources based on expert experiences. These will be discussed shortly. 

This publication provides the field with information on what comprises “quality” 
professional development resources. It provides a set of criteria and indicators to (1) help program 
and professional development staff select and evaluate professional development materials and 
resources, and (2) help professional developers and researchers prepare and develop quality materials 
and resources. The criteria are not intended to be exhaustive of all the factors that professional 
development staff should consider in evaluating every resource. However, they can serve as a 
template-a foundation-to facilitate thinking about the advantages and disadvantages of a particular 
resource. 

Below we discuss how the criteria were developed, how they are organized, and how they 
can be used. 
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How Were The Critema Developed? 

The criteria were developed using a field-based research approach that included a selected 
literature review, field surveys, interviews, and reviews by experts in the field. PRO-NET staff 
conducted a selected literature review on professional development, with an emphasis on identifying 
resources that are “proven” to be effective. As noted above, the research literature does not focus on 
resources per se. However, several studies do identify key elements of effective professional 
development and have implications for identifying quality resources. Staff also looked at U.S. 
Department of Education criteria for identifying exemplary programs in a number of educational 
areas (e.g., career-technical education programs). Such criteria describe components of exemplary 
programs that have been identified by expert panels in the specific subject area. Components 
included program quality, educational significance, evidence of effectiveness, and replicability/ 
usefulness to others-all applicable to quality resources. Staff also reviewed literature on document 
design and e-leaming to gather information about readability and formatting. In addition, PRO-NET 
staff interviewed a select number of staff developers in programs across the country to identify 
criteria used in developing and selecting professional development resources. 

Once the literature review and interviews were completed, staff synthesized the findings and 
developed a set of draft criteria and associated indicators. The initial draft was e-mailed to over 300 
members on PRO-NET’s mailing list. The criteria were revised based on comments received. Next a 
“Request for Field Input” was posted on the PRO-NET 2000 Web site which included the revised 
criteria and a series of structured questions. The revised criteria were then sent to a panel of four 
experts in the field for review and comment. Two sequential conference calls were held with the 
panel to review and refine the criteria. 
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The literature review and interviews in the field identified key elements of quality 
professional development resources based on expert experiences. Below we discuss each of these 
elements. 



How Are the Criteria Organized? 

Twenty criteria were identified through the research conducted. They fall into four distinct 
categories: Appropriateness of General Content; Appropriateness of Design and Delivery; Quality of 
Research Base; and, Ease of Adoption/ Adaptation. Below is a brief description of each of the 
categories. Each category has a set of criteria as well as indicators or descriptors that more fully 
explain the criteria. The criteria appear in Appendix A. 

Appropriateness of General Content 

The content of professional development must be based on the systematically identified 
needs of instructors as well as the goals of the program. Needs assessments are the foundation of all 
professional development activities as they raise the level of individual and program awareness 
regarding (1) areas of strength, (2) areas for improving instruction, (3) individual learning 
preferences, and (4) preferred approaches to professional development. Professional development 
will be more effective if its content is related to the needs of the instructors relative to their programs 
and learner populations. (See “Professional Development Resources Supplement: Improving 
Instruction, Organization and Learner Outcomes Through Professional Development” for a 
discussion of guidelines and strategies for conducting needs assessments, http://www.pro- 
net2000.org). Content is often identified by both the instructor’s self-determined needs and 
preferences and the program administrator’s needs to improve instructional services, correct a 
program deficiency, implement a program change, and meet a Federal or State mandate. 
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Content also must be aligned with national, State, or district standards in core content 
academic areas (e.g., reading, mathematics, and English language acquisition). As States seek to 
improve program quality by providing a strong foundation in academic skills for all learners, 
professional developers will need to design a program that is coherent with these standards. To aid in 
the selection of professional development resources, it would be valuable to the field if the authors of 
the training materials specify, at the outset, the State or district content standards to which the 
materials apply. 

Once the content is determined, it is necessary to select the resources that will support the 
professional development. Educators may be able to choose from existing resources, or they may 
have to develop new materials. In either case, the following questions will help them to assess the 
content: 

□ Is the content aligned with the learning goals? 

□ Is the content aligned with the State and/or district standards? 

□ Does the content contribute to or enhance the knowledge base? 

Appropriateness of Design and Delivery 

In addition to content knowledge, quality professional development is based on knowledge of 
how adults learn. Adult learner theory recognizes that adult learners bring prior knowledge and a 
wealth of experiences to the learning environment. Instructor backgrounds, levels of motivation, 
knowledge and experience, and work environments impact learning and the ability to change as a 
result of professional development. Effective professional development builds on this knowledge and 
provides experiences with complex, real-world problems and situations. It also incorporates a variety 
of learning modalities and accommodates a variety of learning styles. The literature identifies several 
core elements associated with the design of effective professional development-it must be focused on 
content knowledge, be coherent with other practices and reforms, and promote active learning (Garet, 
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et. al). It also must be structured so that it is sustained over time (not a one-shot workshop); allow 
opportunities for practice and feedback; and provide opportunities for reflection. 

There are a variety of ways to deliver professional development that incorporate these 
elements. These may include (1) a series of workshops on a specific content area that incorporates 
theory, demonstration, practice, feedback and coaching (Joyce and Showers, 1988) with a facilitator 
that has the appropriate content knowledge (Loucks-Horsely, et al., 1987); (2) peer coaching or 
mentoring sustained over time with opportunities for practice and feedback (Joyce and Showers, 
1995); (3) inquiry research that is embedded in instructors own teaching practices (Cochran-Smith 
and Lytle, 1992); and (4) involvement in program improvement and curriculum-development 
activities (Joyce and Showers, 1988). Regardless of the delivery method, the materials used in 
professional development must support these elements. The following questions will help to assess 
the quality of the instructional-related elements: 

□ Are instructional strategies appropriate to target audiences? 

□ Are materials culturally and ethnically sensitive, free of bias, and reflect diverse 
audiences? 

□ Are the goals and objectives clear, challenging, and appropriate for the audience? 

□ Do the materials include a discussion of how new skills and knowledge can be 
applied to individual learning environments? 

Quality of Research Base 

The focus in education is on “what works” in improving learner outcomes as identified 
through research-based studies. The No Child Left Behind legislation set a standard for research 
studies that apply rigorous, systematic, and objective procedures to obtain reliable and valid 
knowledge. Adult education also has placed an emphasis on research to build a stronger foundation 
of knowledge. Professional development staff need to critically assess the quality of the research 
studies that support practices and programs to improve learner outcomes. Questions to ask to meet 
the “gold” standard of scientifically-based research include: Is the research design experimental or 
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quasi-experimental? Are the data reliable and valid? Has the study appeared in a peer review 
journal? Has the study been reviewed by external experts? 

While the ideal is to base selection of programs and practices on rigorous research, the field 
currently lacks such evidence. Therefore, it is essential to have other kinds of information upon 
which to base decisions. For example, when selecting training modules, it is important to know if the 
materials have been field-tested and if there has been some documentation of that field test. Also 
critical is whether there was an evaluation that provided some documentation of the impact of the 
professional development. Although the literature consistently identifies evaluation as a critical 
component in the delivery of professional development to adult educators, it is currently a weak link. 
(See the PRO-NET publication, “Evaluating Professional Development: A Framework for Adult 
Education” for a discussion of evaluation strategies; http://www.pro-net2000.org). Evaluation must 
be incorporated into the professional development process in order to document the changes in 
instructor behavior, program services, and student outcomes resulting from the professional 
development activities. Without such an evaluation it is difficult to determine if the professional 
development, and the accompanying resources are effective and are doing what they are supposed to 
do. Materials need to help the users understand what the assessment components look like. When 
selecting or developing resources educators should ask the following questions: 

□ Has a research study been conducted that applied rigorous, systematic, and objective 
procedures to obtain reliable and valid knowledge? 

□ Is the content knowledge-based? 

□ Has the module been field-tested with target audiences in actual context? 

□ Does the module contain an evaluation plan that is linked to training objectives? 

□ Do the materials provide some evidence of effectiveness? 

Ease of Adoption/ Adaptation 

Once professional development staff determine that training materials support the goals and 
objectives of the program, meet instructor needs, and are aligned with content standards, they are 

ready to consider replicating the materials. In fact, the ease of module replication is an important 
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factor that trainers consider when deciding whether or not to adopt or adapt training materials. 
Contributing factors include the degree of financial costs necessary to secure and/or reproduce 
materials, the time allotment required by staff to secure, adapt, prepare, and set up the resources, and 
the availability and ease of acquiring the required materials. Materials that are burdensome because 
they are difficult to find or purchase, expensive, or that involve extensive preparation by the staff 
before or during use are less likely to be selected by professional development staff who have little 
time and money to devote to resources. It has also been found that the actual document design, or 
structure, itself has a significant effect on the way a reader responds to a resource. Research indicates 
that knowing how readers will use a document is a critical aspect of design (Benson, 1985). Readers 
should be able to find the information they need in a timely maimer. If a document is poorly designed 
or lacks retrieval aids, readers will stop using the document (Ryan, n.d.). Similarly, if the production 
value of a training video is poor, it is not likely to be an effective learning tool. Research also shows 
that when readers cannot understand text or technology, they often blame themselves more than they 
should (Schriver, 1997). The following questions will help to assess the ease of adoption and 
adaptation of resources; 

□ Can the materials or training curricula be replicated in a variety of classrooms and 
learning environments? 

□ Is the information well organized, easy to understand, and easy to use? 

□ Are the materials well written? 

□ Is the layout aesthetically pleasing to the reader? 

□ Is the text legible and easy to read? 

□ Are references correctly cited? 

□ Do the production values of the video and audio images enable audiences to 
understand the content? 

□ Are the principles of Web design followed? 

How Can These Criteria be Used? 

The intended audience of this publication is adult education professional development staff 

and program administrators at the national, State, regional, and local levels. The professional 

development staff is generally responsible for assessing program and instructor needs and planning a 
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systematic approach to professional development. They choose the resources and materials that 
support the professional development or often collaborate with others to develop materials 
themselves. Therefore, they need to be aware of the criteria that constitute quality professional 
development resources. Administrators support the professional development in a variety of ways 
and also should be aware of the criteria for quality resources. Professional development staff can use 
the criteria to rate the quality of different resources. While we recommend all the criteria for these 
purposes, we recognize that it may be difficult to justify every indicator. However, consider using as 
many of the criteria as can be effectively applied to your selection of professional development 
resources. The framework included in this publication covers the four areas described above. Exhibit 
A provides a sample framework that can serve as an example of a user-friendly system to assess 
materials. 

Once professional development staff has identified instructor needs and program goals, they 
can use the chart to assess the quality of different resources related to the area of need. The above 
assessment tool, or use of a Likert-type scale that rates materials, for example, as Consistently High, 
Some Evidence, or Poor will help professional development staff have a better understanding of how 
the resource lines up with what have been identified as key characteristics of quality materials. The 
California Adult Literacy Professional Development Project (CALPRO) developed a rubric that uses 
a four-point rating scale-Exemplary, Adequate, Marginal, and Unacceptable-to help State staff 
assess training modules and is a good resource as a guideline for developing your own rubric. (See 
Appendix B.) 
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Use of such tools can help professional development staff and administrators make an 
informed decision on which resources to select. This activity may be done collaboratively or by 
individual professional developers. In addition, professional developers and researchers can use the 
criteria as a guideline for developing new resources. It is useful for researching and outlining a plan 
to develop materials and as a way to monitor the development of the materials. 

The criteria also can be a basis for professional development activities. Professional 
development staff can work with instructors, or instructors can work independently to review 
professional development resources and materials. This activity encourages instructors to critically 
reflect on the materials they use in their own practices. Alternatively, professional development staff 
can ask instructors to construct a set of indicators based on the criteria provided. These activities are 
a means for introducing the criteria to staff and for developing a shared understanding of the criteria. 

In summary, the guidelines within this publication are intended to enhance the selection and 
development process used when assessing the quality of professional development resources. The 
criteria are intended to help professional developers and administrators assess resources and make 
informed choices about professional development materials. 
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APPENDIX B 



Rating Descriptors for Evaluation of Adult Education 

Training Modules 



The following is a rubric for evaluation of training modules 
developed by CALPRO, A Project of the California Department 
of Education in March 2002. 
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